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(57) Abstract 



An apparatus for controlling a plurality of devices in response to biopotentials produced by gestures made by a user. The apparatus 
includes a sensor (22) applied to the user for producing an input signal representing the biopotentials, wherein the input signal changes as 
a function of changes in the biopotentials. A plurality of bandpass filters (28, 30, 32) are responsive to the input signal, and each of the 
filters produces respective frequency signals. A processor (38) is responsive to the frequency signals from each of the filters and generates 
control signals in response to gestures made by the user to commande operation of a device (40). A method is also provided that analyzes 
the high frequency signals with respect to a plurality of gesture models to detect a plurality of gestures being made by the user and then, 
provides control signals to a device (40) for controlling the device in response to a plurality of gestures made by the user. The method 
includes detecting a short eyebrow lift, a long eyebrow lift, a jaw clench, a short eyebrow drop, right and left eye glances and multiple eye 
blink gestures of the user. In another aspect of the invention, the method provides continuous analog control signals in response to eye 
gestures of the user. 
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P RAIN-BODY ACTUATED SYSTEM 

Field of the Invention 

This invention relates generally to the area of nonmanual human control of external systems. 
More specifically, the invention provides a control system with which a user may control an external 
5 device with control signals produced in response to a collective manifestation of electroencephalographic 
("EEG"), electromyographic ("EMG"), electrooculargraphic and other biopotentials. 

Background of the Invention 

Examples of apparatus and methods for controlling devices exclusively with brain and body bio- 
potentials are known and described In U.S. Patent Nos. 5,474,082 and 5,692,517. In those embodiments, 

1 0 a user wears a headband containing three sensors for detecting brain and body signals at the forehead. The 
signals are transmitted to an "interface unit" where they are amplified, digitized and decoded to produce 
multiple command signals that users can learn to modulate consciously to control machines. It is possible 
with this system to achieve control of multiple functions. 

However, the use of this approach has identified the need for additional capabilities. To make 

1 5 this technology more accessible and useful for a greater number and diversity of individuals, there is a 
need for the graphical user interface (GUI) and the requisite brain and body control skills to become more 
intuitive, comprehensible and easily mastered. There is a further need for new control signal channels 
for users with special needs and functional limitations. Further, novel methods are needed to identify, 
analyze, and apply new easy-to-produce signal characteristics and signal patterns for cursor control. For 

20 example, for an implementation of a hands-free mouse, methods to achieve left and right mouse button 
clicking, cursor speed and resolution switching, and activation and deactivation of the hands free 
mouse/keyboard controller are needed. In addition, methods by which the user can achieve user- 
adjustment of interface sensitivity, mouse-click and switch activation threshold, and cursor offset/bias; 
activation of alternative cursor horizontal motion control for wide field-of-view displays; and the 

25 triggering of keyboard commands such as "tab" and "enter" are also needed. 

Finally, a series of training programs, including a new music program making use of complex 
auditory and visual biofeedback, are needed to help users learn multiple discrete and continuous control 
signal manipulation. 

Summary of the Invention 

30 The present invention provides a comprehensive, new, flexible and sophisticated control system 

that is heretofore unknown. The brain-body actuated control system of the present invention has the 
advantage of improving the efficiency and extending the capabilities of persons engaged hands intensive 
activities. The brain-body actuated control system of the present invention has the further advantage of 
assisting and providing additional capabilities to persons having one or more mental or physical 

35 disabilities. The brain-body actuated control system of the present invention has the advantage of being 
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relatively inexpensive and useable to provide both continuous and discrete control signals in response to 
detecting user gestures. 

In accordance with the principles of the present invention and the described embodiments, the 
invention provides an apparatus for controlling a plurality of devices in response to biopotentials produced 
5 by gestures made by a user. The apparatus includes a sensor applied to the user for producing an input 
signal representing the biopotentials, wherein the input signal changes as a function of changes in the 
biopotentials. A plurality of bandpass filters are responsive to the input signal, and each of the filters 
produces respective frequency signals. A processor is responsive to the frequency signals from each of 
the filters and generates control signals in response to gestures made by the user to command operation 
1 0 of the plurality of devices. 

In another embodiment, the invention includes a method of controlling a device in response to 
biopotentials produced by gestures made by a user. The method requires detecting biopotentials at a 
location on a user and filtering the biopotentials with a high frequency bandpass filter to produce high 
frequency signals in a bandwidth of from approximately 70 Hz to approximately 3000 Hz. The method 
1 5 analyzes the high frequency signals with respect to a plurality of gesture models to detect a plurality of 
gestures being made by the user and then, provides control signals to the device for controlling the device 
in response to a plurality of gestures made by the user. 

In one aspect of the invention, the method provides discrete control signals in response to 
detecting a short eyebrow lift, a long eyebrow lift, a jaw clench, a short eyebrow drop, right and left eye 
20 glances and multiple eye blink gestures of the user. In another aspect of the invention, the method 
provides continuous analog control signals in response to eye gestures of the user. 

In a further embodiment of the invention, the brain-body actuated system is linked with a music 
synthesizer to produce music in response to gestures made by the user. 

The brain-body actuated system of the present invention is especially useful in those applications 
25 where the user's hands are already busy. For example, for those who are performing intensive text or data 
entry, the brain-body actuated controller of the present invention can be used to execute the functions of 
a mouse, thereby permitting the user to keep their hands on the keyboard. In other applications, a user's 
hands may be performing other tasks, for example, an assembly, manufacturing, cooking, cleaning or any 
other task; and the brain-body actuated controller can be used to perform other functions, run appliances, 
30 etc. 

Similarly, in mouse intensive applications, for example, CAD applications, the brain-body 
actuated controller of the present invention can be used to execute various function keys, thereby not 
distracting the user from the mouse activity. For those who do not wish to, or are incapable of, using their 
hands, the brain-body actuated controller can be used in conjunction with voice recognition systems for 
35 simplifying the voice recognition application. In addition, the brain-body actuated controller has numerous 
applications to assist those that have one or more mental or physical disabilities in that it allows those 
persons to perform functions that would otherwise be beyond their capabilities. 
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The above and other objects and advantages of the present invention shall be made apparent from 
the accompanying drawings and the description thereof. 

Brief Description of the Drawings 

Fig. 1 is a schematic block diagram illustrating the brain-body actuated system in accordance with 
the principles of the current invention. 

Figs. 2A-2C is a flow chart of a routine executed by the brain-body actuated system of Fig. 1 that 
provides continuous analog control of a device in response to user's gestures. 

Fig. 3 is a flow chart illustrating a routine executed by the brain-body actuated system of Fig. 1 
that provides further continuous analog control of a device in response to user's gestures. 

Fig. 4 is a flow chart of a subroutine executed by the brain-body actuated system of Fig. 1 that 
provides discrete control signals in response to user gestures. 

Figs. 5 A and 5B is a flow chart of a routine executed by the brain-body system of Fig. 1 that 
provides further discrete control signals in response to user gestures. 

Fig. 6 is a schematic block diagram of a music synthesizing system responsive to user gestures 
in accordance with the principles of the present invention. 

Figs. 7A-7F are graphical representations of the outputs of the three frequency channels of the 
brain-body actuated system in response to different user gestures. 

Petailed Description of the Invention 

Referring to Fig. 1, as described in U.S. Patent Nos. 5,474,082 and 5,692,517 which are hereby 
incorporated in their entirety herein, a brain-body actuated control system 20 receives signals from sensors, 
for example, electrodes, 22 in contact with the forehead of a user 24. The sensors detect a collective 
manifestation of electroencephalographic ("EEG"), electromyographic ("EMG"), electrooculargraphic 
and other biopotentials, hereinafter collectively referred to as biopotentials. The forehead is selected as 
the primary measurement site for the derived control signals. For users with special needs, the forehead 
is often the last site to suffer degradation due to injury or disease. Connection to the forehead is easy, 
requiring minimal skin preparation. The forehead is rich in neural and muscular enervation. 

Alternate measurement sites have also been developed to increase the efficacy of our system for 
users with particular neuromuscular functional limitations and in certain operational settings where the 
forehead site is inappropriate. These sites include: 1) the right and left temple, for example, dry contact 
sensors mounted on the inner surfaces of the forward sections of the eyeglasses' or safety glasses' ear 
pieces, with the reference ground mounted on or built into the right and left nose pieces; 2) the right and 
left cheekbone, for example, similar eyeglass sensor mounts as temple site, with armatures or extensions 
off the ear pieces, same ground; and 3) behind the left and right earlobes and below the mastoids, for 
example, sensors mounted in rear sections of ear pieces of eyeglasses or safety glasses. 
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In the known control system described in U.S. Patent Nos. 5,474,082 and 5,692,517, control 
signals resulting from a combination of bio-potentials for a plurality, for example, ten, control channels 
were derived from user selected reference frequencies residing within respective channels. In contrast, 
referring to Fig. 1, the combination of bio-potentials are now filtered with analog circuitry and processed 
5 in an interface circuit 26 into three separate channels defined by the range of their frequency bands. 

The interface circuit 26 has an amplifier 27 that receives the signals from the electrodes 22, and 
the output of the amplifier 27 is divided by bandpass filters 28-32 into the three separate channels. A low 
frequency channel bandpass filter 28 has break points that provide a bandwidth of frequencies from 
approximately 0.2 Hz to approximately 3.0 Hz. A mid frequency channel bandpass filter 30 has break 

1 0 points that provide a bandwidth of frequencies from approximately 0.8 Hz to approximately 45.0 Hz; and 
a high frequency channel bandpass filter 32 has break points that provide a bandwidth of frequencies from 
approximately 70.0 Hz to approximately 3000 Hz. Each filter break point is achieved with a linear four 
pole filter network. Additionally, the high frequency channel filter 32 further includes an envelope 
detection circuit that rectifies the high frequency channel signal so that it falls within a zero to a plus power 

1 5 value such that the derived signal is proportional to the RMS power of the signal. 

The outputs of the filters 28, 30, 32 pass through a three channel, 12 bit, A/D converter 
34 and are converted from a parallel to serial signal form in the microprocessor 36. The digitized serial 
signal is provided to a computer or processor 38 which handles the GUI interface and decodes and 
processes the biopotential signal from the processor 36 to provide the desired control of the audio visual 

20 display 40. The processor 38 can be any type of processor that is capable of running the "WINDOWS 95" 
operating system. As will be appreciated, the A/D converter 34 can alternatively be placed to receive the 
output signal from the amplifier 27; and the three frequency channel bands are provided by digitizing the 
forehead signal and then decomposing the resulting digitized signal into various frequency bands. 

The low range is predominantly sensitive to eye motion and blinking. The mid range spans the 

25 accepted EEG frequency bands and is sensitive to a variety of facial and eye movements as well as various 
brain resonance such as alpha wave and beta wave generation. The high frequency range selectively 
responds to muscle activity such as subtle contractions of the masseter and frontalis muscles. 

Brain and body biopotential signals derived from a user's forehead and other sites provide signal 
attributes and wave forms suitable for activating mouse left and right button clicks, for example, to move 

30 a cursor 42 in a vertical direction 44 and a horizontal direction 46 on the display 40, keyboard characters, 
and other computer commands and software functions. Through appropriate loop closure and 
multidimensional feedback, users are able to perform a repertoire of facial muscle, EEG-based and eye 
glance discrete actions which can be efficiently identified by state machine discrimination algorithms. 
These discrete actions are defined as user "gestures". Any number of gestures may be utilized to create 

35 discrete action commands for control of a device, for example, a cursor; and such discrete action command 
gestures are later described in detail. However, using cursor control as an example, various methods of 
control in response to different gestures are first described. 
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Two-Axis Cursor Motion Formulae and Methods 

The following methods were developed to improve and expand x-y axis cursor control and to 
extend it's efficacy to a wider diversity of individuals in a broader range of operational settings: 

1 - Step- Velocity Cursor Motion 

For both vertical and horizontal motion, cursor speed is varied with a look-up table of graduated 
velocities in accordance with the current magnitude of the frequency of control; and such magnitude is 
calculated using a moving average time window as described in U.S. Patent Nos. 5,474,082 and 5,692,517. 
This approach provides a computationally economical multi-speed cursor method, practical for 
implementation in a mouse driver, to improve user control feedback, cursor accuracy/resolution, and 
interface ease of use. 

2 - Low Freq uency Channel Horizontal Cursor Motion Method A 

The cursor is programmed to move to the right as a direct function of the time-averaged 
magnitude of the low frequency channel. Within the low frequency resonance band, magnitude increases 
are sensitive to eye movements. When the low frequency channel's magnitude drops below a user-defined 
lower threshold, the cursor moves to the left automatically in accordance with a user-adjusted, 
programmed acceleration. Such a user-adjusted acceleration control is described below with respect to 
the Auto-Drop Bi-Polar Cursor Motion. This method improves two-axis control by fostering greater 
separation between x and y-axis control signal response. Eye motion and blinking create rightward cursor 
motion by producing increases in low frequency signal power while not significantly affecting power in 
higher frequency channels mapped to vertical cursor motion. 

3 - low Frequency Channel Horizontal Cursor Motion Method B 

This method is based on a left-right phase reversal observed in the wave form of the low 
frequency channel signal corresponding to direction of lateral eye motion. This method uses the shape 
rather than the power of the low frequency signal to determine the direction and velocity of cursor motion 
on the x-axis. A sharp glance to the left and then forward produces a characteristic wave form in the low 
frequency channel which initiates leftward cursor motion. A sharp rightward and then forward glance 
produces a mirror-reversed waveform, which initiates rightward cursor motion. In the described 
embodiment of the invention, a state machine algorithm, depicted in Fig. 2, is used to look for the presence 
of these two patterns to determine when a left glance or right glance has occurred and produce subsequent 
right or left cursor movement. The outputs of the three frequency channels in response to a look right 
glance are illustrated in Fig. 7 A. 

This algorithm cursor velocity is a direct function of the absolute magnitude of the initial peak 
of the low frequency left/right glance waveform. The user waits until a "ready" cue in the WINDOWS 
taskbar indicates the low frequency waveform has returned to a defined Ready State before further control 
inputs can be issued. 

To increase the effective cursor speed, this method provides an alternative configuration in which 
a second or third glance in the same direction and immediately following the first glance causes the cursor 
to continue moving in the triggered direction even before the control signal returns to the Ready State. 
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This method also facilitates greater separation between x and y-axis response to user control 
inputs and is very easy to understand and intuitive to use. Due to the wave form specificity of this method, 
spurious or unintended horizontal cursor responses to random eye movements are inhibited, such as those 
that might occur in certain work environments or operational settings with wide-screen displays. 

5 Prior to implementing the algorithm of Fig. 2, various default and user controlled settings are 

made and counters and switches are initialized. The user can choose to use default settings for high and 
low frequency value thresholds (HfvThres and LfvThres) to start with. Default values are as follows: 
HfvThres=0.5, LrvThres=0.4, StasisCnts (time within thresholds)= 8sec , and TransCnts(time to cross from 
one threshold to the other )=1.2sec . 

10 The algorithm of Fig. 2 is iterated approximately every 10 milleseconds C*ms"). The state 

machine is initially set to State=l 1; or alternatively, the state machine is set to State=l 1 at 202 if the high 
frequency value is detected to be equal to or greater than the high frequency threshold at 203. The 
State=l 1 is detected at 204 of Fig. 2B. To activate a left or right gesture, the user has to first keep their 
Low frequency value (Lfv) within the plus and minus threshold values (205) for a count equal to 

1 5 StasisCnts (206, 208). Once the user maintains the Lfv at State=l 1 for the StasisCnt, the state machine 
at 210 sets the State=0 and the Ready Display and reset the other display triggers. 

Upon detecting the State=0 at 2 1 1 in Fig. 2 A, the state machine at 212 tests to determine if the 
Lfv exceeds the upper threshold. If the state machine detects a low frequency value excursion in the 
positive direction, the state machine at 214 switches to State=l. Upon detecting State=l at 216, the state 

20 machine at 218, 220, 222 updates an internal accumulator of the low frequency values. If at 224, the state 
machine detects that the low frequency value moves below the negative threshold, the machine at 226 
switches to State=2. At this point, the state machine has detected a positive excursion beyond the positive 
threshold and a negative excursion below the negative threshold. If a State=l counter at 228, 230 expires 
before a State=2 is detected, the state machine is reinitialized at 232, thereby forcing the machine to wait 

25 for the user to bring their Lfv within the two thresholds. 

State=2 is detected at 234, and the low frequency value accumulator is updated at 236-240. The 
state machine then at 242 detects that the low frequency value is at a threshold less than one-half the low 
frequency threshold indicating that the low frequency signal is moving back toward zero. A user gesture 
causing a large positive low frequency value, followed by a large negative low frequency that is followed 

30 by a signal moving toward zero is representative of an acceptable user command gesture. Such a 
command gesture may be a movement of the eyes to the right or to the left, etc. Thereafter, at 244, 
direction and FirstCrossing values are set. The direction value is used at 246-250 to identify the direction 
of motion of the detected user gesture. The identification of the gesture on the display 40 must match the 
user's gesture. It is possible for the identification to be the opposite; and therefore, the state machine 

35 permits the user to bring the machine display into conformance with the user's action. 

The state machine at 244 also sets a FirstCrossing value =1, calculates and stores a low frequency 
value magnitude and resets the low frequency value accumulator to zero. The low frequency magnitude 
value is broadcast to cursor movement program and is used to continue cursor motion. Then, at 252, the 
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state machine switches back to State=ll. Similarly, the state machine returns to State=ll if the low 
frequency value at 242 is not detected within a period of time counted by the State=2 timer at 254, 256. 
The state machine also, referring to Fig. 2C, sends a confirming display message to the user indicating that 
a proper right or left gesture (depending upon what the direction switch is set to) has been detected and 
that the cursor will be moved in the proper direction. 

If the user initiates a gesture in the opposite direction, the negative excursion of the low frequency 
signal is detected at 260; and at 262, the state machine is switched to State=3. State=3 is detected at 264, 
a large positive excursion is detected at 266 of Fig. 2B, after which the machine is switched to State=4 at 
268. State=4 is detected at 270; and at 272, a negative moving excursion is detected. Thereafter, at 274, 
the direction switch is set, a FirstCrossing value = -1 is set, the low frequency value magnitude is 
calculated, stored and provided to the cursor control program. The cursor control program is a lookup 
table in which the algebraic value of control signals are stored in association with desired corresponding 
values representing the direction and velocity of cursor motion. Interpolation between stored values may 
be used in a known manner. The machine is then set to State=l 1 at 276. The state machine also, referring 
to Fig. 2C, sends a confirming display message to the user indicating that a proper right or left gesture 
(depending upon what the direction switch is set to) has been detected and that the cursor will be moved 
in the proper direction. 

When the state machine detects a proper left or right gesture, the FirstCrossing value is set Then 
when the state machine returns to State=l 1 , if the user initiates another gesture in the same direction, as 
detected at 205, 280 and determined by the value of FirstCrossing at 282, 283. In this case, at 284, 285, 
the appropriate state is set and the corresponding state counter is reset, thereby allowing the state machine 
to begin processing a potential gesture. If a gesture in the opposite direction is detected as indicated by 
the sign of FirstCrossing at 286, 287 and Trigger=l at 288, 289, the state machine immediately at 290, 291 , 
zeros LfvMag and sends a Stop message to the user. In this way, a look gesture in the opposite direction 
of the movement the user started will stop the cursor. 

The state machine of Fig. 2 displays the various Low frequency gesture states the user creates 
with the display portion of the code. The continuously updated value LfvMag is sent to the code that 
computes left-right cursor movement. 

4 - Mid Frequency Chan nel Eve Motion Horizontal Cursor Control 

Careful analysis of the mid frequency channel's response to brain-body inputs shows that the 
lower portion of the lower frequency values of this channel are also sensitive to eye motion or glances. 
For example, a right eye glance causes an immediate increase in the magnitudes of any of a number of 
user-selected reference frequencies of the mid frequency channel ranging from .9 Hz to 6 Hz. To create 
an intuitive means of horizontal cursor control, this method generates rightward cursor velocity as a direct 
function of the time-averaged value of a user-selected reference frequency within the lower portions of 
the mid frequency channel. Using the signal processing described in the previously referenced U.S. Patent 
Nos. 5,474,082 and 5,692,517, the magnitude of a control signal is used to generate rightward cursor 
velocity. A null amplitude about a midpoint of a magnitude from an eyes stationary state to an eyes right 
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glance state is determined by a user in a calibration or initialization process or routine. With this eye 
motion cursor control, the magnitude of the control signal is sampled and subtracted from the midpoint; 
and the algebraic difference is used to provide a command to the cursor control program. As previously 
described, the cursor control program is a lookup table of command magnitudes versus cursor direction 
5 and velocity. 

Consequently, with this method, the user looks to the right to move the cursor to the right. 
Keeping the eyes relatively stationary lowers the magnitude of the control signal below a user-defined 
lower threshold and the cursor moves to the left automatically in accordance with a user-adjusted, 
programmed acceleration. 

1 0 In this method, a discrete left glance gesnire, as described in paragraph 4 of Discrete Command 

Gestures, can be used to stop cursor motion until the state machine returns to the Ready State. This creates 
stability in the control interface and prevents the counterintuitive eventuality of leftward eye glances 
causing rightward cursor motion. 

As with methods two and three, this method improves two-axis control by fostering greater 

1 5 separation between x and y-axis control signal responses. Rightward eye motion creates rightward cursor 
motion by increasing power in the lower portion of the mid frequency channel while leaving power in the 
high frequencies (mapped to vertical cursor motion) essentially unaffected. 
5 - Mid Frequency Channel Mental-Based or Non-Muscle Vertical Cursor Control 

This method uses a real-time online analysis of signal 'chaos' or time-averaged degree of 

20 variability of magnitude samples at specific reference frequencies of the mid frequency channel to initiate 
vertical cursor motion. A focus of attention on a perceptual cognitive task presented on the display, 
concentration on visual imagery or mental rotation, and listening to complex and periodic music stimuli 
all have been shown to evoke an increase in the variability or 'spread' of certain signals within the mid 
frequency channel. The signals in these channels correspond to an upper portion of the mid frequency 

25 channel as detected on the forehead/prefrontal site. These reference frequencies are at 35 Hz and/or 40 
Hz. 

Signal spread is defined as the variability of the sampled magnitude values over a sliding sample 
window which is fully described in the previously referenced U.S. Patent Nos. 5,474,082 and 5,692,517. 
The cursor moves up as a direct function of increase in signal spread above a user-adjusted multiple of a 
30 baseline value and down in accordance with a user-adjusted programmed acceleration when the signal 
spread drops below a user-adjusted multiple of a baseline value. The baseline value is calculated 
individually for each user during an online signal-sampling period in which the user is instructed by screen 
text and audio to "relax and think of nothing in particular." 

This energy efficient method provides an intuitive vertical cursor control option that responds 
35 selectively to mental activity and offers the control method of last resort for certain users with severe 
neuro-muscular limitations. 
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6 - Mid Frequency Channel Muscle-Based Vertical Cursor Control 

This method capitalizes on the fact that certain high frequency signals within the upper portion 
of the mid frequency channel from approximately 12 Hz to approximately 45 Hz are sensitive to muscle 
activity, for example, frontalis muscle activity, that results in a slight eyebrow lift as well as brain activity. 

5 Such activity causes an increase in the magnitude of the derived lock-in signal magnitude. 

For those situations in which easily learned intuitive control is desired, a high frequency reference 
signal of the mid frequency channel (ranging from approximately 12 Hz to approximately 25 Hz) is 
mapped to the up and down movement of the cursor. 

Using the signal processing described in the previously referenced U.S. Patent Nos. 5,474,082 

1 0 and 5,692,5 17, a control signal magnitude from the mid frequency channel is provided. The cursor attains 
an upward velocity as a direct function of the time- averaged magnitude of a user-selected frequency within 
an upper portion of the mid frequency channel. A null amplitude about a midpoint of a magnitude from 
an eyebrows relaxed state to an eyebrows lift state is determined by the user in a calibration or 
initialization process or routine. The magnitude of the control signal is sampled and subtracted from the 

1 5 midpoint; and the algebraic difference is used to provide a command to the cursor control program which 
provides the desired up or down cursor velocity. In this way, if the user raises their eyebrows, a 
corresponding increase in the magnitude of the derived control signal causes the cursor to move up. 
Relaxation of the eyebrows causes the control signal magnitude to drop below the null amplitude, thereby 
causing a corresponding lowering of the cursor in accordance with a user-adjusted, programmed 

20 acceleration. 

This intuitive mapping of forehead muscle activity to vertical cursor movement fosters separation 
between x and y-axis cursor response (when the horizontal axis is controlled with one of the previously 
described methods) and rapid learning of two- ax is control. 

7 - High Frequenc y Channe) Vertical Control 

25 Through careful analysis, it was learned that subtle contractions of the frontalis, masseter, 

temporalis, and other facial muscles within a certain proximity about the sensor locus, will cause 
biopotential signal level transients most pronounced in the high frequency channel. This signal activity 
is used to provide a rapid, reliable and selective method for control of the vertical axis of the two-axis 
cursor. 

30 As in the method described above, using the signal processing described in the previously 

referenced U.S. Patent Nos. 5,474,082 and 5,692,5 17, a control signal magnitude from the high frequency 
channel is provided. The cursor attains an upward velocity as a direct function of the time-averaged 
magnitude of a user-selected frequency within an upper portion of the high frequency channel. A null 
amplitude about a midpoint of a magnitude from a relaxed state of a gesture, for example, an eyebrow lift, 

35 to an active state of the gesture is determined by the user in a calibration or initialization process or routine. 
The magnitude of the control signal is sampled and subtracted from the midpoint; and the algebraic 
difference is used to provide a command to the cursor control program which provides the desired up or 
down cursor velocity. In this way, if the user makes the gesture, a corresponding increase in the magnitude 
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of the derived control signal causes the cursor to move up. Relaxation of the gesture causes the control 
signal magnitude to drop below the null amplitude, thereby causing a corresponding lowering of the cursor 
in accordance with a user-adjusted, programmed acceleration. 

In this method, upward cursor motion is generated as a direct function of the high frequency 
5 channel signal. Unintentional high frequency channel discrete command gestures (see next section) are 
avoided because easily controlled, very subtle, high frequency signals (well below the gesture thresholds) 
reliably and accurately raise the cursor. When the high frequency channel signal drops below a user- 
selected lower threshold, the cursor moves downward automatically in accordance with a user-adjusted, 
programmed acceleration. 

10 This method creates a highly stable, easily understood, quickly mastered, and accurate cursor 

positioning method. When it is applied to control of the vertical axis there is virtually no corruption of, 
or interference with, the horizontal control signals as described above (methods two through four). 

8 - Radial-Programmed (Polar) Cursor Motion Method 

The above derived control signals (described in methods two through seven) can be used to 
1 5 control cursor motion in a radial, rather than a linear, vector to create a polar cursor control paradigm. In 
this method, the cursor follows a continuous radial motion on the display even when receiving no control 
inputs. At such times, when the control signal is below the lower threshold, the radius of the cursor's 
programmed motion is three pixels. The radius of cursor motion increases as a direct function of the time- 
averaged magnitude of the user-selected control signal. Cursor velocity along the radial path increases 
20 with the radius. 

An alternative method increases the cursor radial velocity as a direct function of the time- 
averaged magnitude of a user-selected control signal suitably contrasting (in its response to brain-body 
inputs) to the signal used to control the radius of cursor motion. 

Any screen location at which a discrete mouse click command is issued (see discrete command 
25 gestures below) becomes the new center point for the programmed radial cursor motion. 

This method affords rapid acquisition of distant screen locations and targets and provides accurate 
and easily mastered cursor positioning in two axes for people limited by their physiology, pathology or 
special circumstances to a single control input. 

9 - Automatic Cursor Motion 

30 An automatic cursor motion technique is employed to facilitate movement in the downward and 

leftward directions. As before, a null zone is defined about a midpoint of the signal magnitude by a user 
initialization process. In the control mode, when the user's signal value drops below the lower end of the 
null zone, which is centered on the midpoint, an automatic cursor motion procedure that is time based is 
initiated. The procedure followed to create the automatic downward or leftward movement is shown in 

35 Fig. 3 which is executed every 100 ms. 

Referring to Fig. 3, prior to executing this procedure, initial values are set The user sets their null 
zone (the signal range centered on the midpoint for which there is no cursor movement), and the various 
drop values. The following default values which are user adjustable are offered to the user to begin: 
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DirPause=1.0sec, SlowTime=1.0sec, SlowSpeed=1000, NormalSpeed=4000, NormalTime=3.0sec, and 
BoostDelta=8000. DirPause is the time the cursor movement is held to zero after a movement reversal 
from positive to negative direction. SlowTime is the time the cursor moves at a slow speed after the 
DirPause time. NormalTime is the time the cursor moves at a gradual increase set by NormalDelta. 
5 NormalDelta = (NormalSpeed-SlowSpeed)*1000/NormalTime. Once the cursor movement reaches 
NormalSpeed, the cursor movement is continuously incremented by BoostDelta until it reaches a 
MaxSpeed=l 0,000. 

As in the prior methods, using the signal processing described in the previously referenced U.S. 
Patent Nos. 5,474,082 and 5,692,517, a control signal magnitude is produced from an output of either the 

1 0 mid or high frequency control channels. A null amplitude about a midpoint of a magnitude from a relaxed 
state to an active state is determined by the user in a calibration or initialization process or routine. The 
null amplitude is defined by negative and positive limits about a zero valued midpoint. Referring to Fig. 
3, as long as the control signal magnitude value is below the negative null limit at 302, the auto drop 
routine proceeds to work through the auto drop sequence implementing the use-created drop times and 

1 5 drop speeds. Further, the time that the magnitude value is below the negative null value is measured by 
incrementing a counter at 304. Next at 306, the counter value is tested with respect to DirPause time; and 
if it is less, a pseudo mag value is set to zero at 308 and sent to the cursor control program to command 
a zero cursor velocity. 

If the counter value is greater than the DirPause time, it is then compared to the SlowTime at 3 10; 
20 and if it is greater than the DirPause time, the pseudo magnitude is set to a slow speed value at 3 12 and 
sent to the cursor control program. A command signal is selected from the lookup table causing the cursor 
to move slowly in the downward direction. In a similar manner at 314, 316, the counter value is tested to 
determined whether the cursor value should be set to a normal cursor speed. If the counter value is 
sufficiently large, a command will be generated at 3 18 to move the cursor at a high speed. 

25 Discrete Command Gestures 

High Frequency Channel Gestures 

In the high frequency channel, five facial muscle-based gestures are used to create discrete action 
commands: 1) a short, concise raising of the eyebrows (short eyebrow lift), 2) a longer eyebrow lift with 
a greater minimum duration (long eyebrow lift), 3) a short, concise, lowering of the eyebrows (short 
30 eyebrow drop), 4) a clenching or closing of the jaw (jaw clench), and 5) three rapid consecutive 
intentional blinks (three blinks). Each of these discrete gestures can be differentiated from the others by 
a pattern-recognizing algorithm and thus mapped to separate mouse, keyboard, or other input commands 
and functions. 
1 - The Short Eyebrow Lift 

35 This gesture is created when the user rapidly and concisely lifts their eyebrows. As illustrated 

in Fig. 7B, this action of the frontalis creates a characteristic voltage spike in the high frequency channel 
of the user's brain-body signal when measured at the forehead site. Users can create this waveform 
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reliably and quickly (Nelson et al.. 1996). There are a number of ways in which this waveform can be 
evaluated to determine if it is an actual short eyebrow lift gesture. 

In the described embodiment, the derived high frequency channel power signal is evaluated with 
a state machine illustrated in Fig. 4. The state machine at 402 compares the signal value to threshold. 

5 When the signal exceeds the threshold, a threshold timer called Timer is started at 404; and with each 10 
ms iteration through the program, the Timer measures the time that the value remains above the threshold. 
When the magnitude of the high frequency magnitude value drops below the threshold, the state machine 
determines that the Timer is running at 406 and then increments the Timer at 408. Thereafter, at 410, a 
minimum click width is checked. The minimum click width defaults to zero, but the user may select a 

10 width value or time as required by the user. Thereafter, a below threshold timer called BIThT is started 
at 412. At 414, the below threshold timer is tested with respect to an after pulse width time. If the below 
threshold tinier is equal to the after pulse width time at 414 and the Timer is equal to or less than the click 
width plus the after pulse time at 4 16, a pulse high frequency waveform is registered as a short eyebrow 
lift gesture at 418. Then, at 420, 422, a short eyebrow lift is displayed to the user to confirm that the 

1 5 computer has recognized that gesture. 

If, during an iteration, the signal goes above the threshold as detected at 402, before the below 
threshold timer reaches or exceeds the after pulse width, then at 404, the Timer continues and the below 
threshold timer is reset. The following default values are provided to the user to start: Thresholds, 
MinClickWidth=0, AfterPulseWth=0.12sec, and ClickWidth=0.3sec. The threshold levels, derived signal 

20 sensitivity, maximum pulse width, and after pulse width are all user-adjustable. 

If at 416, it is detected that the signal went above the threshold value and then down after a time 
that is greater than click width plus the after pulse width, that is, the maximum pulse width, then one of 
two gestures are being made, that is, a long eyebrow lift or a jaw clench. A more comprehensive 
discussion of these two gestures is contained in paragraphs 2 and 3 below. Further, it has been determined 

25 that a long eyebrow lift is also reflected in the low frequency channel. If the long eyebrow lift is being 
made, as shown in Fig. 7C, the low frequency channel will be in some state other than a relaxed or Ready 
State; and therefore, the low frequency value ready signal state of Fig. 2 will be false. The jaw clench has 
no impact on the low frequency analysis as shown in Fig. 7D; and therefore, with that gesture, referring 
to Fig. 2 at 210, the low frequency value ready signal state will be true. In Fig. 4, at 424, a test of the low 

30 frequency ready signal is made. If it is true, the low frequency signal is within the positive and negative 
thresholds; and therefore, the high frequency gesture is identified as a jaw clench at 425. A jaw clench 
is identified to the user at 426, 428. Similarly, if low frequency ready signal is false, the long eyebrow lift 
gesture is identified at 430; and a long eyebrow lift is identified to the user at 432, 434. 

Fig. 4 also includes a debounce function if activated by the user. A debounce switch is initially 

35 defaulted to, and detected as, zero at 436. When a gesture is identified at 418, 425, 430, the timer is set 
to zero and the debounce switch is set to one at 438. Thereafter, with each iteration, the debounce timer 
is incremented at 440; and the timer is compared to the user selected debounce time at 442. When the 
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debounce time expires, the debounce switch is reset to zero at 444. Thus, the debounce process is effective 
to ignore unwanted repeated discrete actions or gestures that are unique to a particular user. 

The short eyebrow lift is easily learned and natural to execute. To facilitate this learning process, 
a graphical display showing the time history of the high frequency channel signal superimposed over the 

5 threshold level is provided. In this way, the user is able to monitor their eyebrow lift action relative to the 
threshold By working with the display, the user is able to adjust the threshold level and sensitivity of the 
high frequency channel signal by increasing or decreasing the amount of eyebrow lift effort required to 
produce a software-recognized short eyebrow lift gesture. It should be noted that such threshold and 
sensitivity adjustments can be utilized with all of the gestures as well as the different modes of two-axis 

10 control. 

If the user has excessive ambient or spasmodic muscle activity in the high frequency band, they 
are able to set the threshold value above this ambient "noise". This is accomplished by adding an offset 
to the user's signal, which has the effect of lowering the signal relative to the threshold. This has been 
found to be necessary for users with Multiple Sclerosis and Cerebral Palsy, for example. The user is also 

15 provided with the capability to manually trigger a computer-simulated short eyebrow lift gesture 
waveform. The form of mis simulated gesture is determined by the user-adjusted threshold, maximum 
pulse width and after pulse width. This graphical image provides the user with a target pulse shape that 
must be achieved for successful triggering of a short eyebrow lift gesture. The user then trains to achieve 
this shape and/or adjusts the state machine settings so that the pulse shape they desire to produce is 

20 acceptable by the computer as a short eyebrow lift gesture. It should be noted that such an adjustment can 
be utilized with all of the gestures as well as the different modes of two-axis control. 

2 - The Long Eyebrow Lift 

This gesture is created when the user lifts their eyebrows, holds them in an elevated position for 

a time greater than the maximum pulse width, and then lowers them. As illustrated in Fig. 7C, this action 
25 of the frontalis creates a characteristic series and pattern of pulses in the high frequency, mid frequency 

and low frequency channels of the user's brain-body signal when measured at the forehead site. Users can 

learn to create this waveform reliably and quickly. 

The state machine algorithm can look for a characteristic series of peaks, troughs and plateaus 

in the mid frequency channel time history wave form and a coincident change in the low frequency 
30 channel and a coincident signature plateau in the high frequency channel time history. In the described 

embodiment, the state machine uses the low frequency channel information of Fig. 2 and the high 

frequency channel information of Fig. 4. 

3 - The Jaw Clench 

This gesture is produced when the user contracts or clenches their jaw muscles. As illustrated 
35 in Fig. 7D, this action of the jaw contraction produces a very large transient in the high frequency channel 
of their brain-body signal as measured at all the sites of measurement discussed above. The jaw clench 
waveform has similar attributes to the short eyebrow lift waveform. It produces a pulse or a series of 
pulses that can be actuated to exceed a predefined threshold level. A key difference between the short 
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eyebrow lift and jaw clench gestures is that the eyebrow lift gesture can normally be executed quickly, 
sometimes as fast as .02 seconds. The jaw clench gesture is typically longer in duration, normally around 
0.8 seconds. 

4 - The Short Eyebrow Drop 

5 This gesture is created by a rapid and concise lowering of the eyebrows or the action of drawing 

the eyebrows together. As illustrated in Fig. 7E, this generates a unique combination of waveforms in the 
high and mid frequency channels, which are detected by a state machine discrimination algorithm. This 
algoridun looks for a high frequency pulse which is similar to, but slightly smaller than, the voltage spike 
defining a short eyebrow lift gesture and a concurrent peak in the mid frequency signal, which is phase 

1 0 reversed from the mid frequency trough typically accompanying the short eyebrow lift. 

5 - The Three Blinks 

Referring to Fig. 7F, tliis gesture of three rapid, consecutive volitional blinks creates a unique 
pattern of three evenly spaced small pulses in the high frequency channel time history which is 
discriminated by a simple state machine algorithm. 
15 Mid Frequency Channel Gestures 

6 - The Brainfinger gesture 

This gesture is created from a user-selected, derived reference frequency in the mid frequency 
channel and triggers a discrete gesture command when its magnitude exceeds and then falls below a user- 
adjustable threshold. This method provides the alternative of a non-muscle-based discrete command 
20 gesture for users with severely limited facial muscle functionality. The same state machine as used for the 
detection of high frequency gestures in Fig. 4 is used for detection of a Brainfinger gesture. User 
threshold, signal sensitivity, and pulse width adjustments are made as with the high frequency gestures 
described above. 

Low Frequency Channel Gestures 

25 7 - The left glance and right glance 

These gestures of swiftly glancing to the left and then back forward create a unique pattern in the 
low frequency channel which is discriminated by a state machine algorithm and is defined as a left glance 
gesture. The action of swiftly glancing to the right and then back forward creates a unique pattern in the 
low frequency channel, called a right glance gesture, which is phase reversed from the left glance gesture 

30 pattern. The two gestures are discrirninated by the same state machine algorithm, which is shown in Fig. 
5 and which is similar to the state machine of Fig. 2. The same initial values and counters that are used 
for the Left-Right-Look Cursor Control State Machine of Fig. 2 are used in Fig. 5. 

Referring to Fig. 5A, the state machine first tests at 502 whether there is high frequency signal 
activity that exceeds the high frequency threshold. If activity is detected/ the state machine at 504 is set 

35 to the wait state of State=l 1 and the StI lCnt counter is forced to zero. With each successive 10 ms 
iteration, the State^l 1 is detected at 506 of Fig. 5B; and at 508, the state machine detects if low frequency 
value stays within the positive and negative thresholds. The State=l 1 counter is incremented at 510 and 
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its count is compared to the stasis count value at 5 12. When the stasis count value is reached, the State=0 
is set at 514. 

Once in State=0 as detected at 5 1 8 of Fig. 5 A, the state machine activates a search for any 
right/left glance waveforms in the signal. While in the Ready State, if the signal first exceeds the positive 
5 threshold at 520, the state machine will move into State=l at 522. State=l is detected at 524; and at 526, 
the state machine determines if the low frequency signal then reverses and exceeds the negative threshold. 
If so, the state machine moves into State=2 at 528. State=2 is detected at 530; and the state machine 
detects at 532 if the low frequency value reverses again and moves to a value = -LfvThres+0.1. That is, 
has the low frequency value reversed; and if so, a gesture is identified at 534, 536 in accordance with the 

1 0 state of the direction switch at 538. Thereafter, the state machine is returned to State=l 1 at 539. This must 
occur within the appropriate transition time (TransCnts) as detected at 542 for the state machine to register 
that a right or left glance (depending upon the setting of DirSw) has been detected. Likewise, if from the 
Ready State, the state machine detects at 544 that the low frequency signal crosses the negative threshold, 
the state machine moves through State=3 and State=4 in a manner similar to that previously described. 

1 5 The polarity of the low frequency signal is a function of the electrode placement relative to the 

generated bio-potentials. Therefore, the user is given the ability to reverse the polarity of the presented 
low frequency signal to insure their right or left glance action correctly matches their intended cursor 
direction command. This is saved as the variable DirSw. The user is also given the ability to set the levels 
of the positive and negative signal thresholds, the low frequency signal sensitivity, and the state machine 

20 left/right glance discrimination timings (TransCnts and StasisCnts). Presenting a continuously running 
time history display of the low frequency channel signal with the two thresholds superimposed over it, 
facilitates the training of the look gestures. Computer-simulated left and right glance waveforms are 
provided to give the user visual targets to emulate for successful left/right glance generation. 

To make the eye glance detection state machine more robust, coincident activity in mid and high 

25 frequency channels can be monitored as well. In the described embodiment, if the high frequency energy 
(Hfv) goes beyond a threshold level (HfvThres), it is interpreted as an indication that some forehead 
activity other than a simple eye glance has occurred. For example, an eyebrow lift can produce a similar 
waveform in the low frequency channel but it concurrently produces a spike in the other two channels as 
well. If this occurs, the state machine resets and waits in State=l 1 for the signal to return to a Ready State. 

30 The user is provided with this information (i.e., eyebrow lift cues) during training so that they can know 
what it 'feels like' to be in and out of the Ready State. 

All the above discrete command gestures, when initiated by the user, are accompanied by visual 
cues in the WINDOWS taskbar. The visual cues can be user-formatted as either text or pictographs of faces 
performing the appropriate gesture. Auditory WAV file gesture cues in the form of various short, easily 

35 identified bell-like musical tones are also provided. 
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Gestures and Cursor Motion 

In this approach, all the discrete gesture commands described above arrest cursor motion for a brief user- 
adjustable interval of time (typically one second) regardless of any concurrent x or y-axis cursor 
positioning control inputs the user may be intentionally or unintentionally sending. This creates 
5 predictable and stable cursor response. When users initiate discrete gestures to make selections or send 
keyboard characters or other command inputs, the cursor remains over the selected icon, button, or other 
active area of the display. This method also facilitates greater control accuracy because gesture commands 
can be used to stop the cursor immediately upon acquisition of intended targets. 

Mapping Of Discrete Command Gestures 

1 0 In use, the discrete command gestures are employed to provide full left and right mouse-button 

click and click-and-drag functionality, enhance the intuitive control of the computer or machine interface, 
and to improve the pointing accuracy of the cursor control. In the described embodiment, the short 
eyebrow lift gesture provides a quick, easy and reliable left mouse button click command. When it is 
executed, the two-axis cursor ceases all motion for a user-adjustable interval (typically one second), a left 

1 5 mouse button command is sent to the computer's operating system, and a "left click" cue appears in the 
WINDOWS desktop task bar. If a double left mouse button click is required, the user simply performs 
a second short eyebrow lift gesture within a user-adjusted time limit after the first click gesture command. 
Click-and-drag commands can be sent to the operating system by executing a third short eyebrow lift 
gesture in a similar manner. For tasks requiring extensive click and drag operations, the mapping of a jaw 

20 clench, for example, is mapped to a click and drag command. 

In cases where frontalis functionality is limited or absent, or where forehead command action is 
inappropriate, the Brainfinger gesture, jaw clench, or eye right/left glance gestures can be user-formatted 
to produce the mouse button clicks. 

The combination of a left glance gesture followed by a jaw clench gesture can be user-formatted 

25 to trigger the system in and out of "sleep mode". Sleep mode is defined as the state in which cursor 
movement inputs and discrete command gestures do nothing to alter the behavior of the system other than 
the toggling of sleep mode off. The discrete gestures can also be formatted individually to generate 
keyboard ASCII character commands and character strings, i.e., a "Tab" character command when the user 
looks right and an "Alt-Tab" string when the user looks left. 

30 The ability to toggle between different mappings of forehead signal magnitudes to cursor x and 

y speeds can be accomplished with jaw clench gestures and eyebrow drop gestures. In a typical 
configuration, toggling between high cursor speed and low cursor speed is accomplished with the jaw 
clench gesture. The low speed provides a high resolution at the expense of speed and the high speed 
provides a fast moving cursor at the expense of accuracy. The eyebrow drop gesture is used to step down 

35 the high speed incrementally with each eyebrow drop gesture. 
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Because each application may require unique mapping of discrete gestures to computer or 
machine action, the invention is set up so that the user and/or the application can specify the desired 
gesture-mapping configuration. 

The above brain-body actuated system provides a number of both discrete and continuous control 
signals in response to gestures of the user, thereby providing a comprehensive, new, flexible and 
sophisticated control system that is heretofore unknown. Such a control system is especially useful in 
those applications where the user's hands are already busy. For example, for those who are performing 
intensive text or data entry, the brain-body actuated controller of the present invention can be used to 
execute the functions of a mouse, thereby permitting the user to keep their hands on the keyboard. In other 
applications, a user's hands may be performing other tasks, for example, an assembly, manufacturing, 
cooking, cleaning or any other task; and the brain-body actuated controller can be used to perform other 
functions, run appliances, etc. 

Similarly, in mouse intensive applications, for example, CAD applications, the brain-body 
actuated controller of the present invention can be used execute various function keys, thereby not 
distracting the user from the mouse activity. For those who do not wish to or are incapable of using their 
hands, the brain-body actuated controller can be used in conjunction with voice recognition systems for 
simplifying the voice recognition application. 

The brain-body actuated controller has numerous applications to assist those that have one or 
more mental or physical disabilities in that it allows those persons to perform functions that would 
otherwise be beyond their capabilities. 

Three Channel, Multi-Signal Brain-Body Actuated Music Program for Training of Multiple Discrete 
and Continuous Control Signal Manipulation 

To foster user understanding of, and a sense of causal connection to, multiple discrete and 
continuous control signals, a new brain-body driven musical synthesis program is used. The same signal 
acquisition, power analysis, and wave form discrimination methods as described above and in the 
previously cited U.S. Patent Nos. 5,474,082 and 5,692,517 are used. This method employs complex 
multi-layered auditory and visual biofeedback to reward harmonic coherence, volitional patterns, and/or 
periodicity in various user-selected, derived reference frequency bands of the mid frequency channel with 
consonant, dynamic and/or pleasing music accompaniment. Incoherent signals, or a bio-spectral pattern 
lacking in energy in the beta, and/or gamma bands, are negatively reinforced with an inert or otherwise 
uninteresting musical accompaniment. 

A schematic diagram of the music synthesis program is illustrated in Fig. 6. In the music 
synthesis program, the mid frequency signals of the bio-spectral analysis software are used to calculate 
time-averaged magnitudes of an array of user-selected mid frequency channel reference frequencies 
ranging from approximately 7 Hz to approximately 45 Hz. This is done in the brain resonance signal 
generation module 602. In the described embodiment, user-selectable reference frequencies are restricted 
to a certain repertoire of values. Values that when transposed up by five octaves (multiplied by 32) into 
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a range that is more audible to the human ear, can be selected to correspond to frequencies of musical notes 
of the Western Chromatic Scale (i.e., A, A#, B, C, C#, D, D#, etc.). These are defined as "brainwave 
resonance frequencies." The time-averaged magnitudes at the user-selected brainwave resonance 
frequencies are compared to user-adjusted threshold levels called "trigger points" in a resonance trigger 

5 module 604. The trigger points for each selected brainwave resonance can be set individually. 

Values of the average signal magnitude values and trigger status are sent to the accompaniment 
module 606 in which corresponding user defined song templates are activated according to the information 
received. When any selected brainwave resonance's average magnitude exceeds its trigger point, it is 
activated. The accompaniment module 606 then sends musical commands, for example, MIDI commands, 

10 to display module 608 that includes the PC computer's sound card or an external sound module 
corresponding to the transposed 'pitch' of the brainwave resonance (i.e., a beta, brainwave resonance of 
13.75 Hz generates the note "A" which is 440 Hz or 5 octaves above). Alternatively, a short repeating 
sequence of pitches centered around the pitch corresponding to the brainwave resonance frequency, called 
a "loop," (i.e., the musical phrase: "A, B, A, G, A, A, A") can be triggered. A loop will continue to play 

1 5 (be activated) as long as its corresponding brainwave resonance frequency exceeds its trigger point. 

User-selection of the brainwave resonance frequencies to be used for triggering musical events, 
the corresponding musical pitches and loops to be triggered, as well as user-formatting of the notes 
comprising the loops is accomplished in the composer module 610. This can be done in one of three ways. 
In one method, the compose module 610 presents a selection of possible musical pitches and "loop 

20 presets" (short musical phrases, of progressive complexity, centered around a primary musical note 
corresponding to a selected brainwave resonance frequency) displayed on a musical Grand Staff. The 
Grand Staffs appearance and functionality is in the manner of current standard music notation software 
programs such as Finale™ or Cakewalk™. The user then chooses from among these alternatives to setup 
the musical material to be activated for each selected brainwave resonance frequency. In this way, a 

25 custom "song template" is created. 

The second method uses the compose module's Grand Staff display to provide the user a means 
to mouse-click individual musical pitches and pitch values (in the manner of current standard music 
writing software programs) to construct custom loops for brainwave resonance triggering. 

The third method involves a loop ''record" module in which the user employs their high frequency 

30 channel derived RMS signal power to affect the vertical motion of a selection-cursor along a vertical scale 
incremented with the notes of the Western Chromatic Scale (displayed in Standard Musical Notation) 
through the melody module 612. By modulating their high frequency channel signal, the user can cause 
the cursor to "dwell" or remain in a stationary position at desired pitch levels for a user-defined minimum 
. selection time interval ("dwell-time"). In this way, the user can construct each loop to be used in the song 

35 template by selecting each component loop pitch in a hands-free manner. The duration of each loop to 
be 'recorded' or formatted in this way is user-defined. After 'recording* each loop, the user is given the 
opportunity to accept or reject it as part of their overall constructed song template. 
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The instrumentation (program assignment or voice) for each brainwave resonance frequency can 
be separately formatted by the user with a pictographic musical instrument selection graphical user 
interface. The mouse click is used to increment the musical instrument pictograph (and corresponding 
instrument) forward through each selection on the instrument menu. In the described embodiment; brass, 

5 reeds, percussion, upper strings, lower strings, bass, guitar, piano, flute, vocal, bells, harp, ambient pads, 
and sound effects instruments are all provided in the instrument menu. 

Any number of different song templates can be produced. Once a song template has been created, 
it can be selected from a play menu. As the music program plays the brain-body activated audio output; 
a visual display consisting of a virtual keyboard is sent to the PC computer video. The virtual keyboard 

10 emulates an overhead view of a piano keyboard. As brain resonance frequencies exceed their trigger 
points, the corresponding musical notes (transposed up five octaves) are highlighted on the virtual 
keyboard. 

In addition, a kaleidoscopic pattern called a "mandala" is sent to the PC video display 608. The 
mandala pattern "paints' color-coded diamond shapes in a radial symmetric, recursive pattern in 

15 accordance with the brainwave resonance frequencies currently exceeding their trigger points. The 
diamond's color is determined by the brainwave resonance frequency: the lower frequency brainwave 
resonance frequencies are color-coded red, the low-middle: yellow, the middle: green, the high: blue and 
the highest: violet (in the manner of the familiar "roygbv" of the visual light spectrum). Thus, the red 
diamonds begin to appear on the display when their corresponding red brain resonance frequency has 

20 exceeded its activation threshold. They appear at a radius corresponding to the current magnitude of the 
corresponding triggered brainwave resonance frequency. They 'linger* in the mandala pattern for a brief 
timed interval after the triggering brainwave resonance has dropped below its threshold. The size of the 
diamond is incremented in accordance with the user-selected magnitude level of the trigger point of the 
corresponding brain resonance frequency. 

25 An improvised melodic line can be superimposed over the mid frequency channel brainwave 

resonance accompaniment during the playing of a song template through the melody module 612. The 
method is similar to the method for hands-free loop formatting ('recording') described above. The high 
frequency channel derived RMS signal power drives the melody module 612 by raising and lowering a 
selection-cursor and thus the pitch level or increment of a vertical musical scale. The musical scale is 

30 displayed in Standard Musical Notation and is user-selected from a menu of major, minor, and pentatonic 
scales and modes. In the described embodiment, the pentatonic (five-note) scale is employed due to its 
harmonic compatibility with a wide range of harmonic accompaniments and key centers. 

Typically, user's can raise the melody pitch incrementally with subtle eyebrow lifts or gentle 
pressure of the tongue against the roof of the mouth. The melody pitch is lowered upon relaxation. 

35 Increasing tongue pressure on the roof of the mouth is a technique familiar to instrumentalists such as 
flautists and trumpeters who use it to raise the pitch of their instruments. 
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Low frequency and high frequency channel discrete gestures can be user-formatted as command 
inputs to create real-time changes in various parameters of the melody and accompaniment such as 
melodic scale transposition, stereo pan, musical instrumentation changes, and other musical effects. 

A left glance gesture, as described herein, can be used to send a left pan command of one 
5 increment to the display module to create a stereo pan effect. A right glance gesture can be used to pan 
one increment back to the right. A jaw clench gesture can be used to transpose the melodic scale of the 
melody module 612 up a user- formatted interval. This discrete high frequency channel gesture can be used 
to create larger interval skips and leaps in the continuous high frequency derived RMS power generated 
melodic line. A short eyebrow lift can be used to sw itch the program assignment (instrumentation) of the 
10 melody module to a new instrumentation value 612. 

Thus, the musical synthesis program as described provides a new way to create and play music 
using new controls derived from gestures of the user. Such a music synthesis program has been found to 
be very pleasant and relaxing to the user. Further, such a program is useful in music therapy in that it 
envelopes the user with auditory and visual displays that provide a compelling experience of self 
1 5 discovery. 

While the present invention has been illustrated by a description of various embodiments, and 
while these embodiments have been described in considerable detail, it is not the intention of Applicants 
to restrict or in any way limit the scope of the appended claims to such detail. Additional advantages and 
modifications will readily appear to those skilled in the art. For example, in the described embodiments, 
20 the detection and identification of gestures of a user from a filtered composite biopotential signal is used 
to control different devices such as a mouse and keys of a computer and a music synthesizer. As will be 
appreciated, the present invention may be used to control other devices such as switches on appliances, 
communications equipment and other devices in residential, business or manufacturing environments and 
on vehicles of all types. 

25 In the described embodiment the brain-body actuated system includes three bandpass filters. As 

will be appreciated, depending on the application, all three of the filters may not be required. For example, 
if only discrete control signals are required as are produced by the algorithm of Fig. 3, then only a high 
frequency filter is necessary. Further, the low mid and high filters have bandwidths that have been 
determined to be particularly effective. However, as will be appreciated, other bandwidths may be used, 

30 and the brain-body actuated system will still detect gestures as described herein. While the limits of the 
bandwidths are susceptible to variations that are an order of magnitude different from those recited, they 
may be varied by a factor of two or three and still be operable. 

Thus, the invention in its broader aspects is not limited to the specific details, 
representative apparatus and method, and illustrative example shown and described. Accordingly, 

35 departures may be made from such details without departing from the spirit or scope of Applicants' general 
inventive concept. 

What is claimed is: 
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1 . An apparatus tor controlling a plurality of devices in response to biopotentials produced by 
gestures made by a user comprising: 

a sensor applied to the user for producing an input signal representing the biopotentials, the input 
signal changing as a function of changes in the biopotentials; 

a plurality of bandpass filters, each of the bandpass filters being responsive to the input signal for 
producing respective frequency signals; and 

a processor responsive to the frequency signals from each of the filters for generating control 
signals in response to gestures made by the user for commanding an operation of the plurality of devices. 

2. The apparatus of claim 1 further comprising: 

a first bandpass filter providing a bandwidth of frequencies from approximately 0.2 Hz to 30 Hz; 
a second bandpass filter providing a bandw idth of frequencies from approximately 0.8 Hz to 45 

Hz; and 

a third bandpass filter providing a bandwidth of frequencies from approximately 70 Hz to 3000 

Hz. 

3. An apparatus for controlling a cursor in a display of a computer in response to biopotentials 
produced by gestures made by a user comprising: 

a sensor applied to the user for producing an input signal representing the biopotentials, the input 
signal changing as a function of changes in the biopotentials; 

a plurality of bandpass filters, each of the bandpass filters being responsive to the input signal for 
producing frequency signals; 

a display; and 

a computer responsive to the frequency signals from each of the filters for generating control 
signals in response to gestures made by the user for commanding motion of the cursor. 

4. A method of controlling a device in response to biopotentials produced by gestures made by a 
user comprising: 

detecting biopotentials at a location on a user; 
filtering the biopotentials to produce frequency signals; 

analyzing the frequency signals with respect to a plurality of gesture models to detect a plurality 
of gestures being made by the user; and 

providing a control signals to the device for controlling the device in response to a plurality of 
gestures made by the user. 

5. The apparatus of claim 4 further comprising analyzing the frequency signals to detect a short 
eyebrow lift. 
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6. The apparatus of claim 4 further comprising analyzing the frequency signals to detect a long 
eyebrow lift. 

7. The apparatus of claim 4 further comprising analyzing the frequency signals to detect a jaw 
clench. 

5 8. The apparatus of claim 4 further comprising analyzing the frequency signals to detect a short 
eyebrow drop. 

9. The apparatus of claim 4 further comprising analyzing the frequency signals to detect three eye 
blinks. 

10. The apparatus of claim 4 further comprising analyzing the frequency signals to detect one of a 
1 0 short eyebrow lift and a jaw clench. 

11. The apparatus of claim 4 further comprising analyzing the frequency signals to detect one of a 
short eyebrow lift, a long eyebrow lift and a jaw clench. 

12. A method of controlling motion of a cursor in a display of a computer in response to biopotentials 
produced by gestures made by a user comprising: 

1 5 detecting biopotentials at a location on a user; 

filtering the biopotentials with a low frequency bandpass filter to produce high frequency signals 
in a bandwidth of from approximately 0.2 Hz to approximately 3 Hz; 

analyzing the low frequency signals with respect to a plurality of gesture models to detect a 
plurality of gestures being made by the user; 
20 providing control signals to the display of the computer to control motion of the cursor in 

response to the plurality of gestures detected. 

13. The apparatus of claim 12 further comprising analyzing the low frequency signals to detect an 
eye glance to the right. 

14. The apparatus of claim 12 further comprising analyzing the low frequency signals to detect an 
25 eye glance to the left. 

15. The apparatus of claim 12 further comprising analyzing the low frequency signals to detect 
relative eye motion between two limits of eye motion. 



I 
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16. Tlie apparatus of claim 12 further comprising analyzing the low frequency signals to detect no 
motion. 

17. A method of controlling a device in response to biopotentials produced by an eye glance gesture 
made by a user comprising: 

5 detecting biopotentials at a location on a user; 

filtering the biopotentials with a low frequency bandpass filter to produce low frequency signals; 

analyzing the low frequency signals with respect to an eye glance gesture model to detect an eye 
glance gesture being made by the user: and 

providing a control signal to operate the device in response to detecting the eye glance gesture 
10 being made by the user. 

18. The method of claim 1 7 further comprising detecting a series of excursions of the low frequency 
signals in opposite directions. 

19. The method of claim 1 8 further comprising: 

detecting the low frequency signals moving in a first direction and exceeding a first threshold; 
1 5 detecting in response to the low frequency signals exceeding the first threshold, the low frequency 

signals moving in an opposite direction and exceeding a second threshold; 

detecting in response to the low frequency signals exceeding the second threshold, the low 
frequency signals moving in the first direction and exceeding a third threshold; and 

providing a signal identifying the eye glance gesture in response to detecting the low frequency 
20 signals exceeding the third threshold. 

20. A method of controlling a device in response to biopotentials produced by a short eye brow lift 
gesture made by a user comprising: 

detecting biopotentials at a location on a user; 

filtering the biopotentials with a high frequency bandpass filter to produce low frequency signals 
25 in a bandwidth of from approximately 70 Hz to approximately 3000 Hz; 

analyzing the high frequency signals with respect to a short eye brow lift gesture model to detect 
a short eye brow lift gesture being made by the user; 

providing a control signal to operate the device in response to detecting the short eye brow lift 
being made by the user. 

30 21. The method of claim 20 further comprising detecting excursions of the high frequency signals 
in opposite directions for periods of time. 



WO 00/33731 PCT/US99/29398 



-24- 

22. The method of claim 2 1 further comprising: 

detecting the high frequency signals moving in a first direction and exceeding a first threshold; 

detecting in response to the high frequency signals exceeding the first threshold, the high 
frequency signals moving in an opposite direction and dropping below the first threshold; 
5 measuring the time that the high frequency signals continuously exceed the first threshold; 

measuring the time that the high frequency signals are continuously below the first threshold; 

providing a signal identifying the short eye brow lift in response to the high frequency signals 
continuously exceeding the first threshold for a first period of time and the high frequency signals being 
below the first threshold for a second period of time. 



WO 00/33731 PCT/US99/29398 

1/15 




r 



22 



Sensor 



42 



46- 



Audio Visual 
Displays 



INTERFACE 



Amplifier 



27 ' 



Low frequency channel 
band pass filter 



28 



r 



34 



Mid frequency channel 
band pass filter 



High frequency channel 
band pass filter 



Analog to 

digital 
converter 



32 



36 



Micro- 
processor 



Computer 



40 



38 



26 
J 



Figure 1 



WO 00/33731 



2/15 



PCT/US99/29398 



Enter Routine with Low freq. value (Lfv) and High freq. value (Hfv) every 0.01 sec 



Yes 



Hfv>=HfvThres? 
203 



^—2 sYes 

— K^State=0? ">-l 



— ►<^State=1?^> 

Yfesn ^218 



—*<r State=2? J> < Lf 
YeS I 224>\ 



No 



202 



State=11 


Stop=False 


St11Cnt=StasisCnts/2 


Wait=True 


Trigger=0 


Ready=False 


Right=Fa!se 


Left=False 



212 



211 




I2_ 



Lfv >LfvThres? 




Yes 



r 



216 



No 



260 



r 



214 



State= 1 

St1Cnt=0 

LfvArea=Lfv 



Lfv < -LfvThres? 



No T ' Y< 

-~< J_fv < 0?^ > 



Yes 



Yes 



No 



<State=3 

St3Cnt=0 

LfvArea=Lfv 



^262 



,-222 

>JLfvArea=LfvArea+Lfv l 



LfvArea=LfvArea-Lfv 



220 



-232 



St1Cnt=St1Cnt+1 



/228 ^ c^230 

/ ^C^1Cnt>TransCntsT>- 



No v 



Stop=False 
Trigger=0 
State=11 
St11Cnt=0 



Wait=True 
Ready=False 
Right=Faise 
Left=False 



Jfes 



Lfv < -LfvThres? 



236 
Yes 



No T 'Yes 




No 



State=2 
St2Cnt=0 



^226 



LfvArea=LfvArea-Lfv 



LfvArea=LfvArea+Lfv 



238 



J 



St2Cnt=St2Cnt+1 



X^t2Cnt>TransCnts 
254 



240 258 
■^^256 
Yes 



No 



246 r 



Stop=False 


Wait=True 


Trigger=0 


Ready=False 


State=1 1 


Right=False 


St11Cnt=0 


Left=False 



No 



242 



Yes 



Lfv< -LfvThres/2? 



248 



Yes yXJ. — .No ' 
<DirSw=17> I Right= 



Trigger=1 
FristCrossing=1 

LfvMag=(LfvArea* 1 00-1 000)*1 2 
LfvArea=0 



Right=True 
Left=False 



it=False 
=True 



-250 



Ready=FaIse 

Stop=False 

Wait=Faise 

State=11 

St11Cnt=0 



244 



252 




264 



St3Cnt=St3Cnt+1 



*<St3Cnt>TransCnts? 



<^Lfv<0?>»| LfvArea=LfvArea-Lfv[» 



(A) L-» {rfvArea=LfvArea+Lfv | 
Figure 2A 



No 



\^es 
ntsj» 



Trigg er=0 
State=11 
St11Cnt=0 
Stop=False 



Wait=True 
Ready=False 
Right=False 
Left= False 



© 



WO 00/33731 



3/15 



PCTAJS99/29398 




State=4 
St4Cnt=0 



-268 



© 



Stop=False 
Trigger=0 
Stated 1 
St11Cnt=0 



Wait=True 
Ready=False 
Right=False 
Left=False 



Yes 



St4Cnt=St4Cnt+1 ^ 
LfvArea=l_fvArea+Val 



><^irSw=lY>> 



Right=False 
Left=True 



NotZ>. Right=True 
Left=False 



Trigger=1 

FirstCrossing=-1 

LfvMag=(LfvArea*100-1000)*12 

LfvArea=0 



274 



J 



St11Cnt=0 
State=11 
276 Stop=False 
Wait=False 
Readv=False 



204 



205-x 



—#<^tate=m^ 




Lfv <LfvThres 

And 
Lfv>-LfvThres? > 



280 



I No 



^< Lfv>0? > ^ Yes 




r 



206 



St11Cnt=St11Cnt+1 



St11Cnt>StasisCnts? 



Yes 




FirstCrossing=1? 



Yes 



286 




No 



No 



282 



No 



-<C^Fir^rossing=-j?^ 



r 



284 



State=1 
St1Cnt=0 



210' 



Left=False 

Right=False 

Ready=True 

Wait=False 

Stop=Faise 

State=0 

St0Cnt=0 

Trigger=0 

LfvMag=0 

LfvArea=0 

FirstCrossing=Q 



Yes 



^<^rigger<>0?^> 

1 ^ 

288 No 



Yes 



290 



Trigger=0 Stop=True 
Wait=False Ready=False 
Right=False Left=False 
St11Cnt=StasisCnts/2 



No 



r 



283 




FirstCrossing=-1? 



Yes 



■X^F irstCrossing= 



No 



State=3 
St3Cnt=0 



285 



289 



287 



Yes 



Nop<jngger<>or> — 
^ ^Yes 



® 



291- 



Trigger=0 Stop=True 
Wait=False Ready=False 
Right=False Left=False 
St11Cnt=StasisCnts/2 



Figure 2B 



WO 00/33731 



PCT/US99/29398 



4/15 




Hfv<HfvThres? 



No 




JYes 



>< ^Left=Tme ?^> ►< ^ption<>"l^ 



No 



Caption="Left" 



<^ight=True^> ►< ^ption<> M Righ T^» 



No 



Caption= M Right" 



<^eaay=^^ 




Caption="Ready tt 



Caption<>' 



"Stop^>-^ 



Caption="Stop tt 



< Wait=True ?^> — ►<^ aption<> tt Waif ?^>-^ 



No 



Caption="Wait" 



•><^Caption<>" "?^>-> 



Caption= n " 



Broadcast to rest of Code Left-Right Cursor LfvMag and Look States 



Figure 2C 



WO 00/33731 PCT/US99/29398 

5/15 



Come here with Mid or High frequency magnitude value, "Mag" 
If Mag below negative null zone, value implement auto drop 



302^ No 



No, 1 — .Yes 

^ag<NegNujl>- 



Counter+1 



304 



No ! ▼ Yes 

|-^<^ounter<DirPaus^>- 

1 ^310 



Mag=0 



No 



308 



Yes 



ounter< SlowTime+DirPaus 



r 



312 



Mag=SlowSpeed*DeltaStep 



No 




Yes 



;ounter<NormalTime+SlowTime+DirPause 
314- 




^ * <^16 

p<Speed<NormalSpee<T>i 
Yes -^fJo 



318 



Speed=Speed+NormalDelta 



Mag=Speed*DeltaStep 



Speed=Speed+BoostDelta 

Mag=Speed # DeltaStep 

Counter=Counter-1 



Counter=0 
Speed=0 



!<^§peed>MaxSpee3^> — » Speed=MaxSpeed 

j I 



Broadcast Mag value to be used for resulting cursor movement computation 



Figure 3 



WO 00/33731 



6/15 



PCT/US99/29398 



Enter with High Freq. or Mid Freq. "Value" and LrvReady 



436 



J 



440 



Yes 



DbSw=1? 




442-^ 



444-s 



No 



DbTmr=DbTmr+1 
Timer=0 




DbTmr>=DbTime? 



Yes 




No 



DbSw=0 
DbTmr=0 



-406 



No/" ^\Yes / \No 

-<Value<ThresholdJ> — KTimeroO? , 

402 408 



Yes 



410> 



Timer=Timer+1 




Yes 



Timer=0 



No 



Timer<MinClickWidth? 




r 



412 



BIThT=BIThT+1 



414. 






BIThT=AfterPulseWth?^>— | 
^Yes 

Timer<=ClickWidth+AfterPulseWth? 



BIThT=B!ThT-1 



< <13^ 



No 



418^ 




r 



438 



[No ~— Yes j 

I ►<^uvReady=True^>--> Clench=True 

424^> — -. 1 

M~ I I nnnl i«-Tn it 



Timer=0 
DbSw=1 



BIThT=0 
Timer=Timer+1 



404-^ 



No 
Yes 



LongLift=True 



425 
430 



— ►< ^mer>300^ >> Timer=Timer- 



No 



420 



no \T es 

I — <^ShortLift=True7> ► 

1 A 426 

No/ ^^-xYf- 

• — <M_ongLift=Truer> 
' 1 r 432 



"^22 



Display "ShortLifT 



428 

J 



Display "LongLifT 



-434 



No^ ^ ^Yes 

r4-<^ Clench=Truer>^ 



Display "Clench" 



Broadcast Lift and Clench values to rest of code 



Figure 4 



WO 00/33731 



PCT/US99/29398 



7/15 



Enter Routine with Low freq. value (Lfv) and High freq. value (Hfv) 



502 -—^ * v^Yes 

<^Hfv>=HfvThres?">- 



518 



No 



State=11 


Right=Fa!se 


St11Cnt=0 


Left=False 


Ready=False 





504 



v Yes 



State=0? 



No 



No > 

r*-<S 




Lfv >LfvThres? 



\ 



^524 
T f 

8tate=1?^> 





Yes 



State= 1 

St1Cnt=0 

Ready=False 



522 



Lfv< - LfvThres? 



\Yes 
s? J> ► 



No 



State=3 

St3Cnt=0 

Ready=False 



Yes 



Yes 



St1Cnt=St1Cnt+1 



No 



530 





f r 


St2Cnt=St2Cnt+1 



— <QState=3?N 




State=1 1 
St11Cnt=0 



State=2 
State2Cnt=0 



"528 



State=11 
St11Cnt=0 



> sYes 

><J)irSw=1?^>> 
ia->> — , — ' 



r 



534 



Right=True 



Nol 
536- 



Left=True 



539, 



State=11 
St11Cnt=0 



1 


Yes 

- r 


St3Cnt=St3Cnt+1 




< 



St3Cnt>TransCnts? 

~t — 



Lfv > LfvThres? 
No' 




State=11 
St11Cnt=0 



State=4 
St4Cnt=0 



© 



Figure 5A 



WO 00/33731 



PCTAJS99/29398 



8/15 



No > i — . 

-<^State=4?^> 



1 


r Yes 


r 


St4Cnt=St4Cnt+1 




St4Cnt>TransCnts? 



Yes 



No 




State=1 1 
St11Cnt=0 



— KPirSw=1?>S 



Left=True 



<C$ alue< LfvThres-0. j ?^>~ 



No 



Right=True 



No 



508 



State=11 
St11Cnt=0 



506 



State=11? 




Va!ue< LfvThres xYes 
And 

Value>-LfvThres? 



Yes 




r 



514 



State=0 
StOCnt=0 
Ready=True 



sYes 

Left=True? > p, 



Display 
"Left" Gesture 



_»^ ^ight=True ^ 





sYes 



Ready=True? 




Display "Ready" 
LfvReady=True 



Broadcast Lfv and Lfv Gestures values to rest of code 



Figure 5B 



WO 00/33731 



9/15 



PCT/US99/29398 



Low Frequency 
Signals 



Mid Frequency 
Signals 



High Frequency 
Signals 



II 



602" 



Look Right/Left 
Gesture 
Detection 



614 



I 



Brainwave Resonance 
Signal Generation 



616- 



618 



Hfv Gesture 
Detection 



604 



Resonance Trigger 
Module 



Keyboard 
Hand-mouse 



Hfv Continuous 
Signal Generation 



Instrument Scale Pitch 
Select Select Select 

MELODY MODULE 



IE 



612-^ 



Hand-entry Hands-free-entry 
COMPOSER MODULE 
Loop Template Creation 



606 
L 



Trigger 
Information 



610 



Stereo Pan 
And other 
Left/Right 
Actions 



Loop 

Templates 



ACCOMPANIMENT MODULE 

Resonance-Triggers to Song-Templates 
[Templates: Notes and Loops] 



r 608 



Accompaniment 
Notes/Display 
Information 



Melody 
Notes/Display 
Information 



DISPLAY MODULE 



A uditory : 
Internal Sound Card, 
External Sound Device 



Visual : 
Computer Monitor, 
Projected Screen, and 
Other Light Projection Media 



Figure 6 



WO 00/33731 



10/15 



PCT/US99/29398 




Figure: 7A 



WO 00/33731 



PCIYUS99/29398 



11/15 



SHORT EYEBROW GESTURE 



HfvThreshold 




Mfv 



^ (j^lickWidth+AfterPulseWdth 



Time-*- 



Lfv 




Time - 



, LfvThreshold 



Time . 



^ - LfvThreshold 



Figure:7B 



WO 00/33731 



12/15 



PCT/US99/29398 



LONG EYEBROW GESTURE 



HfvThreshold 




^ - LfvThreshold 



Figure: 7C 



WO 00/33731 PCT/US99/29398 

13/15 



JAW CLENCH GESTURE 



Start of jaw clench 



HfvThreshold 




Mfv 



(}:iickWidth+AfterPulseWdth 




Lfv 




Time- 



, LfvThreshold 



Time- 



^ - LfvThreshold 



Figure: 7D 



WO 00/33731 



PCT/US99/29398 



14/15 



EYEBROW DROP GESTURE 



Hfv 



Mfv 





I jf hTv i nresnoK 




^C^IickWidth 
1 


Time-> 

+AfterPulseWdth 

/W/Sjwv, 




^^^^ ' 

] 


/V Time-* 
. LfvThreshold 




'A/ \ 
!/ \ 






\ /' 
1 / 1 


A l\ /^ v ^ Time ~* 




l ^ - LfvThreshold 
1 



Lfv 



Figure: 7E 



WO 00/33731 



PCT/US99/29398 



15/15 



THREE BLINKS GESTURE 



Hfv 









^ HfvThreshold 








- - — 



Time- 



Mfv 



V 

IS 


If 










V 






yVV^' 

1 1 Time-** 



Lfv 





Tt 


> 






y 


Vv - 


c/- w 





Figure: 7F 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US99/29398 



A. CLASSIFICATION OF SUBJECT MATTER 

!PC(7) :A61B 5/04 
US CL :600/545 

According to International Patent Classification (IPC) or to both national classification and IPC 


B. FIELDS SEARCHED 


Minimum documentation searched (classification system followed by classification symbols) 
U.S. : 600/544-546,587,595 


Documentation searched other than minimum documentation to the extent that such documents are included in the fields searched 


Electronic d 
WEST 


ata base consulted during the international search (name of data base and, where practicable, search terms used) 


C. DOCUMENTS CONSIDERED TO BE RELEVANT 


Category* 


Citation of document with indication, where appropriate, of the relevant passages 


Relevant to claim No. 


X 


US 5,692,517 A (JUNKER) 02 December 1997, Abstract, col. 6 
lines 12-55, col . 7, line. 25-49 


1, 3-11 


A 


US 4,354,505 A (SHIGA) 19 October 1982, entire disclosure. 


1-22 


A 


US 4,228, 807 A (YAGI et al.) 21 October 1980, entire disclosure. 


1-22 


[~H| Further documents are listed in the continuation of Box C. 


[""1 See patent family annex. 




• Special categories of cited documents: 

"A" document defining the general state of the art which is not considered 
to be of particular relevance 

"E" earlier document published on or after the international filing date 

"L" document which may throw doubts on priority claim(s) or which is 
cited to establish the publication date of another citation or other 
special reason (as specified) 

"0" document referring to an oral disclosure, use. exhibition or other means 

"P" document published prior to the international filing date but later than 
the priority date claimed 


"T" later document published after the international filing date or priority 
date and not in conflict with the application but cited to understand the 
principle or theory underlying the invention 

"X" document of particular relevance; the claimed invention cannot be 
considered novel or cannot be considered to involve an inventive step 
when the document is taken alone 

"Y" document of particular relevance: the claimed invention cannot be 
considered to involve an inventive step when the document is 
combined with one or more other such documents, such combination 
being obvious to a person skilled in the art 

document member of the same patent family 


Date of the actual completion of the international search 
12 APRIL 2000 


Date of mailing of the international search report 

/? 10 MAY 2000 


Name and mailing address of the ISA/US 
Commissioner of Patents and Trademarks J> 




Box PCT ^ -7 

Washington. D.C. 20231 / 

Facsimile No. (703) 305-3230 / 


Telephone No. (703) 308-2767 



c*rm pr-iTKA/VtO (second sheet) (July 1998)* 



